feat: API fetcher service + api_json parser for HTTP polling data ingestion - #52
Merged
Conversation
- fetchers/shared: config loader, atomic incoming writer, state persistence (cursor + seen-files), poll loop with exponential backoff - fetchers/api_fetcher: JWT auth with auto-refresh, windowed paginated fetcher (continuous + backfill modes, param_variants for RSL/TSL), entry point, example config.yml, Dockerfile (build ctx ./fetchers) - fetchers/api_fetcher/mock_server: minimal Flask mock with JWT auth and synthetic hourly CML data (POST /login/, POST /refresh/, GET /cml/)
Parses JSON files produced by api_fetcher. A field map (loaded from FIELD_MAP_PATH env var) does longest-prefix matching on the filename stem to route source fields to [time, cml_id, sublink_id, rsl, tsl]. Includes example_field_map.yml for the mock server (RSL + TSL split).
parser/service_logic.py: import parse_api_json_raw; add elif .json branch routing JSON files to write_rawdata (before else: quarantine) parser/main.py: add .json to FileWatcher extensions; add *.json glob in process_existing_files for startup replay docker-compose.dev.yml: adds mock_server (port 5001) and api_fetcher services for local development; mounts shared data volume and example field_map.yml
fetcher.py: in continuous mode, end was computed as start+chunk_hours; with overlap_seconds >= chunk_hours*3600 (as in the example config) the window end equalled the cursor, so the cursor never advanced (infinite loop). Fix: end = cursor_dt + chunk_hours; overlap only shifts start back. parser/main.py: process_existing_files returned early when there were no CSV files, skipping the JSON glob entirely. Fix: guard only the CSV section, always run the JSON section. mock_server/app.py + docker-compose.dev.yml: healthcheck hit GET /login/ which is POST-only; Flask returned 405, urlopen raised HTTPError, check always failed → api_fetcher never started. Fix: add GET /health endpoint, point healthcheck there.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #52 +/- ##
==========================================
+ Coverage 85.86% 86.21% +0.34%
==========================================
Files 35 37 +2
Lines 3269 3409 +140
==========================================
+ Hits 2807 2939 +132
- Misses 462 470 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
service_logic.py: the JSON elif was placed AFTER 'raw'/'data' in name, but all api_fetcher filenames end in _data.json and matched the CSV branch first, making the JSON branch unreachable. Fix: check .json suffix first, before the substring checks. test_service_logic.py: add test_process_cml_file_json_archived covering the repaired JSON branch (patches parse_api_json_raw, asserts write and archive calls). test_api_json_parser.py: new test module for parse_api_json_raw covering prefix matching, RSL/TSL column routing, UTC datetime normalisation, empty-file handling, and error paths (missing field, no prefix match).
…nfrastructure with API fetcher Resolution adds api_json support to the new ParserBundle architecture: - Add load_api_json_bundle() helper in service_logic.py - Wire api_json parser selection in main.py based on PARSER_TYPE env var - Update generate_config.py validation to accept 'api_json' as valid parser type - Import _make_default_bundle in main.py for test compatibility This preserves all api_fetcher functionality (JSON polling, api_json parser) while adopting the pluggable parser infrastructure from csv-generic-parser PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds automated data ingestion from HTTP/JSON APIs via a new
api_fetcherservice that polls external endpoints and writes JSON files to the parser's incoming directory. Includes theapi_jsonparser that handles these JSON files.Architecture
The api_fetcher runs as a separate container, decoupled from the parser. It:
The parser watches the same incoming directory and processes JSON files identically to CSV files.
Components
fetchers/shared/ — Shared utilities
config.py— Load fetcher config from YAMLpolling.py— Generic polling loop with backoffstate.py— Persist last-polled timestamp to avoid re-fetchingincoming_writer.py— Write fetched data to shared volumefetchers/api_fetcher/ — Main fetcher service
main.py— Entrypoint, wires together componentsfetcher.py— HTTP client with retry logicauth.py— Authentication handlers (API key, bearer token, basic auth)config.yml— Example configurationparser/parsers/api_json/ — JSON parser
parse_raw.py— Parse JSON with field-map-based column routingexample_field_map.yml— Longest-prefix matching on filename to route source fields to canonical names (time, cml_id, sublink_id, rsl, tsl)Integration
parser/service_logic.py— Addload_api_json_bundle()helperparser/main.py— Select api_json bundle whenPARSER_TYPE=api_jsonscripts/generate_config.py— Acceptapi_jsonas valid parser typeConfiguration
Add to deploy repo's compose override:
Parser service uses
PARSER_TYPE=api_json.Testing
test_api_json_parser.pycovering field-map routing, UTC normalization, empty files, error pathsfetchers/api_fetcher/mock_server/for integration testingBackward compatibility
No breaking changes. Existing CSV parsers continue to work unchanged. The api_json parser is opt-in via
PARSER_TYPE=api_json.